[WS1][CUDA][Qwen3-Next] C6: GDN decode-step goldens and recurrent/conv state contract (RFC #428 stack 4/4) - #469

• The runner run at f8d9399 built _C from its own clean clone of the ref and checked that
  both rl_engine.__file__ and rl_engine._C.__file__ were inside that clone. The runner,
  the goldens and the tests are unchanged after f8d9399 (610f1b7 touches only the design
  note; the gated branch's OptionalCUDAGuard fix, which came in by merge at the time, touches
  only csrc/cuda/rmsnorm.cu).
• It reported git_dirty: false. Its recurrent and conv sections are identical to the
  earlier run at 1d69312. Provenance records rl_engine_file (inside the clone) and env
  (FLA_USE_FAST_OPS and TRITON_DEFAULT_FP_FUSION, both unset).
• fusion_check: fusion_off_effective: true; it now covers every variant an arm compiles:
  all 12 fusion-off variants carry enable_fp_fusion=False, all 12 fusion-on variants True.
• conv_kernels (the SiLU specializations, 6 per arm): every variant has ex2.approx (2) and
  div.full.f32 (2) in its PTX, and SASS FFMA is 0 in all 12.
• conv_silu (x widened to fp32, fp32 cache and output; fusion on): pre-activation
  mismatches 0 at B = 1, 4, 17, 64. SiLU outputs differing: 3,060 / 8,192, 12,609 / 32,768,
  53,270 / 139,264 and 199,728 / 524,288; at B=64, 170,817 by 1 ULP, 23,201 by 2, 5,298 by
  3–4, 412 by more. Triton's x / (1 + tl.exp(-x)) on the golden's pre-activation values
  differs from the provider in 0 elements at every B; the variant with div_rn and
  libdevice.exp differs from the golden in 0; the two variants that swap only one of them
  match neither side.
• conv_noact_bf16 (activation off, fp32 cache): with fusion on, the bf16-output
  specialization's PTX has fma.rn.f32x2 (4) and its SASS FFMA (4); the fp32-output
  specialization has mul.f32x2 (4), no FFMA, and 0 mismatches. bf16 outputs differing:
  1 / 0 / 4 / 14 (max|diff| 3.91e-3 at B=64); 18 of the 19 have |out| < 0.11, and the largest
  gap in bf16 ULPs is 96, at |out| ≈ 2.4e-7. With fusion off: no FFMA, and 0 mismatches at
  every B.
• Device: NVIDIA B200, capability (10, 0); torch 2.13.0+cu130, triton 3.7.1, vllm 0.30.0,
  Python 3.12.14.

Cluster, same software as above; each run built _C from a clean snapshot of its ref and
refused to run unless rl_engine._C.__file__ was inside it.

• 09b7aeb (the former first merge of the gated branch, at 5eaeca8; this commit no longer exists on the linear branch, its tree equalled 7b30633 + the gated branch at 5eaeca8): full suite
  15 failed, 3117 passed, 923 skipped (4055 collected); no new failures against main, the
  same 2 fixed. It ran on a different B200 node from the baseline's.
  check_gdn_recurrent_golden.py 31 passed (incl. the provider null-block assert),
  test_gdn_state_contract.py 19, test_qwen3_next_norm.py 136, check_qwen3_next_norm_providers.py 9,
  test_qwen3_next_workload.py 7, test_dispatch.py 17, test_rms_norm.py 57 passed /
  72 skipped, test_ws1_gtest_gpu.py 9, test_vjp_fp32.py 50; check_operator ×3
  pass_rate=1.0000. Its runner (the version at 7b30633) gave recurrent and conv numbers
  identical to the run at 5216a6c.
• 5216a6c: the runner's recurrent and conv numbers that design note §4
  quotes (:134-147, :163-167). Its fusion-off arm did not take effect (the in-process knob
  reused the fused kernel); 10e5230 fixed that, so its FMA reading is not used.
• 2d895d0: the fixed runner. Fusion-off arm effective; mismatch counts and
  max|diff| identical with fusion on and off. This is the fusion A/B on the SiLU path (design
  note :189-191).

Local, 16204ab: 2× B200 (sm_100), driver 580.126.20, torch 2.13.0+cu130, triton 3.7.1,
vllm 0.30.0, transformers 5.17.0, flashinfer 0.6.18.post1, Python 3.12.14.

full suite, local main (32b765e): 19 failed, 2940 passed, 923 skipped
full suite, local cefa0d3:        17 failed, 3078 passed, 923 skipped   (no new failures)

Measured at 16204ab against main, on Python 3.10 with the hook versions pinned in
.pre-commit-config.yaml: real pre-commit, no new findings (commits through 7862a12
were also checked with the pinned hooks); mypy --ignore-missing-imports rl_engine/, no new
errors; mkdocs build --strict, 8 warnings, the same 8 as main.

At c0ab21c, the pinned pre-commit (Python 3.10) over every file the four stacked PRs
change against main (pre-commit run --from-ref 32b765e --to-ref HEAD, 36 files) has one
finding: black and flake8 on rl_engine/kernels/ops/pytorch/norm/rms_norm.py. That file
already fails both on main with the same hunks (four blank lines before the first
top-level function after strict_add_rms_norm, inline-comment spacing, one blank line
before class NativeRMSNormOp; flake8 E303 and E302). The stack adds no lint findings.
The commits after c0ab21c change scripts/ws1_gdn_provider_agreement.py, the design
note and, through the gated merge, csrc/cuda/rmsnorm.cu; at 610f1b7 the runner passes
black, isort and flake8 with the pinned settings, and the design note has no trailing
whitespace and ends in a newline.
At 610f1b7: mkdocs build --strict gives the same 8 warnings as main (the identical
set); mypy --ignore-missing-imports rl_engine/ (mypy 2.3.1, Python 3.10, as ci.yml
runs it) gives 56 errors in 16 files on both main and 610f1b7, the same messages apart
from line numbers. The stack adds no lint, docs or type findings.

• A decode-only batch with no speculative decoding returns early into
  _forward_core_decode_non_spec (qwen_gdn_linear_attn.py:1295-1307) because
  VLLM_ENABLE_FLA_PACKED_RECURRENT_DECODE defaults to on (envs.py:1199-1200).
  tests/check_qwen3_next_norm_providers.py asserts that default; it does not assert which
  kernel actually runs.
• VLLM_GDN_DECODE_KERNEL defaults to "cuda", but qwen3_next.py:495 constructs the layer
  with gqa_interleaved_layout=True, which makes _fused_gdn_decode_unsupported_reason
  (qwen_gdn_linear_attn.py:535-554) return a reason. The layer logs a fallback to
  "triton" (:520-523), or raises ValueError if VLLM_GDN_DECODE_KERNEL was explicitly
  set to cuda (:516-519). The fallback was observed in a real engine (B200, TP1 and TP4,
  real checkpoint, enforce_eager, not strict): the startup log shows Falling back to the Triton GDN decode path and GDN decode kernel: triton. That log is not part of this PR.
• The packed kernel is the provider only for decode-only steps. When the step also holds
  a prefill, _forward_core routes the cached decode rows through causal_conv1d_fn (branch
  at qwen_gdn_linear_attn.py:1373, call at :1378-1388) and
  fused_sigmoid_gating_delta_rule_update (split_non_spec at :1408-1412, branch at
  :1493, call at :1497-1512). So this golden matches what rollout runs in decode-only
  steps, not in steps shared with a prefill.
• Under MTP, the packed kernel is not used in steps that carry draft tokens (see "Deliberately
  not in this PR").

This measurement is not in this PR and has not been published; treat it as a reported
observation until the check that produced it is submitted. Design note §5 labels it the same
way (:225-227). It drove the real GDNAttentionMetadataBuilder.build() and _forward_core
of one standalone layer built from the checkpoint's config.json, on B200. Limits:
parameters and cache were synthetic (no weights loaded); single process, no engine, scheduler
or CUDA graph; the test, not the scheduler, built the attention metadata (decode rows first).

For one target decode request with an fp32 recurrent state, three prefill-bearing step
compositions gave identical numbers. The convolution still matched bitwise; the two
recurrent kernels did not:

At TP1 the difference reached a bf16 output within the next 16 plain decode steps in 4 of 20
seeds. With synthetic parameters that frequency does not carry over to real weights; it shows
only that the difference propagates. At TP4 per-rank head counts it shows up in the same
step's output. This is a provider-side batch-composition dependence, outside what this golden
can fix. Whether the golden's target is right in every step depends on the upstream fix:

• [Model] Enable batch-invariant mixed decode and prefill for Qwen GDN vllm-project/vllm#49827 (open, not merged) routes mixed-step recurrent decodes
  through the packed kernel too. Its two commits, applied to 0.30.0, gave a gap of 0 at both
  head counts above; its scheduler part was not tested. It does not change the mixed-step conv
  path (causal_conv1d_fn). Its own validation is on Qwen3.5 (non-interleaved), H100, TP1.
• Disabling VLLM_ENABLE_FLA_PACKED_RECURRENT_DECODE sends every non-speculative step
  through fused_sigmoid_gating_delta_rule_update. In that configuration the measurement
  found the decode row's output and state bitwise equal across step compositions
  (prefill-target rows still differ, a FlashInfer CP issue). This golden's target would then
  have to change, or a second golden targeting that kernel would be needed.

1. Gating is fused, in fp32: beta = sigmoid(b), g = -exp(A_log) * softplus(a + dt_bias),
   with the kernel's threshold branch at 20. HF computes these as separate PyTorch ops.
2. No repeat_interleave: i_h = i_hv // (HV // H), so q/k stay at 16 heads while v has 32.
3. QK norm is x / sqrt(sum(x*x) + 1e-6): not an RMSNorm, not F.normalize. scale is
   applied to q after the norm, and k is never scaled.

Not a transcription:

• softplus: the kernel computes tl.log(1.0 + tl.exp(x)) (fused_recurrent.py:327); the
  golden computes torch.log1p(torch.exp(safe)), where safe is x on the taken branch
  (gated_delta_rule.py:114-116). These round differently.
• The kernel's exp/log become fast_expf/fast_logf when FLA_USE_FAST_OPS=1
  (third_party/flash_linear_attention/ops/op.py:16-25). The goldens model the default.
• Contractions use the repo's 32-wide chunked sum (_chunked_sum, gated_delta_rule.py:76-94)
  rather than the kernel's tree reduction. It fixes the reduction shape per row regardless
  of batch size, which is the argument for L1; L1 itself is established empirically
  (test_golden_is_batch_invariant, check_gdn_recurrent_golden.py:162-188). The same choice
  is why the golden is not bitwise against the kernel. For a last dimension that is not a
  multiple of 32, _chunked_sum zero-pads to the next chunk boundary (:86-91, since 0b41350;
  before that it fell back to a plain .sum); this does not trigger at K=V=128.
• The test feeds raw randn q/k (check_gdn_recurrent_golden.py:85) with
  use_qk_l2norm_in_kernel=True (:110), so the kernel normalises them, as vLLM's decode path
  calls the provider (qwen_gdn_linear_attn.py:1707). The golden's use_qk_l2norm=False mode
  (prefill's convention) is checked only for producing a different result (:405-427); it is
  not compared against a provider.

vLLM's causal_conv1d_update does not check index values even with validate_data=True;
that flag adds only shape and stride asserts and a null_block_id is not None assert (vLLM
causal_conv1d.py:1151-1153, 1188-1201). An out-of-range index there is an unchecked memory
access. This is read from the source, not exercised.

Asserted for the recurrent pair (check_gdn_recurrent_golden.py:152-156): both outputs are
zeros, and both the golden and the provider leave block 0 untouched. For conv, only the golden
is tested (:326-336). The golden treats negative conv indices as inactive
(tests/test_gdn_state_contract.py:54-59), whereas the provider would treat them as real
indices. The design note states the two contracts separately (:100-110).

Product rounding is pinned by test_conv_bf16_products_round_before_fp32_accumulation (CPU)
and by test_conv_provider_preserves_bf16_product_cancellation, which checks that the
provider and the golden agree on an input whose exact product would not cancel.

The test docstring (:301-302) quotes the measured fp32 counts; they are measured, not
asserted.

With an fp32 cache the output is not bitwise. The cause depends on the Triton
specialization, and both mechanisms are on the provider side. Measured by
scripts/ws1_gdn_provider_agreement.py (conv_silu, conv_noact_bf16) on B200 at f8d9399
(design note :169-205).

• With SiLU (Qwen3-Next's configuration and the check file's): the activation.
  • With the activation off and fp32 output, provider and golden agree bitwise at every
    batch size tested, so the convolution itself matches.
  • Triton lowers acc / (1 + tl.exp(-acc)) (vLLM causal_conv1d.py:1085) to ex2.approx
    and div.full.f32. Applying that expression to the golden's pre-activation values
    reproduces the provider bitwise, and the golden equals libdevice.exp with IEEE
    division. Swapping only one of the two reproduces neither side.
  • About 38% of fp32 outputs differ, mostly by 1 ULP. Only values straddling a bf16
    rounding midpoint survive the store: 0 / 0 / 1 / 5, one bf16 ULP each.
  • No contraction is involved: there is no FFMA in these specializations, and
    TRITON_DEFAULT_FP_FUSION=0 changes nothing.
• Without an activation and with bf16 output: FP contraction.
  • This specialization contracts the tap multiply-adds (fma.rn.f32x2 / FFMA); its
    fp32-output twin does not, and matches bitwise.
  • 1 / 0 / 4 / 14 outputs differ (max|diff| 3.9e-3), mostly near-cancelling ones, where the
    gap reaches up to 96 bf16 ULPs; the count drops to 0 with TRITON_DEFAULT_FP_FUSION=0.
  • Qwen3-Next's decode path does not use this specialization.
• Open: the recurrent provider computes exp(g) and sigmoid(b) with Triton, and the
  golden with PyTorch; the same kind of difference may account for part of the recurrent
  output mismatches. Not investigated (design note :207-210).

Production does not use a conv bias. Qwen3-Next's conv1d is built with bias=False
(qwen_gdn_linear_attn.py:418-421). The only provider comparison without a bias is the single
hand-built cancellation input (check_gdn_recurrent_golden.py:465-475, activation=None).
bias=None with random inputs is never compared against the provider. dim_first=False is
compared only golden against golden (:430-448).

test_golden_is_batch_invariant (check_gdn_recurrent_golden.py:162-188) runs one batch of
64 (seed 3, all rows active) for both state dtypes and checks that rows 0, 1, 31 and 63 produce
bitwise the same output and state block when run alone. This compares the golden against
itself. No L1 claim is made for the provider; the mixed-step observation above reports a
counterexample for vLLM 0.30.0 (mixed decode+prefill steps, standalone layer, synthetic
parameters).

The strict profile uses an fp32 recurrent state; bf16 is a differential experiment (design
note :262-263). test_bf16_state_rounding_stays_within_fixed_fixture_bound (:194-225) runs
the golden against itself (fp32-state run vs bf16-state run), with no provider. The gate
inputs a/b/A_log/dt_bias are fixed for all 128 steps; only the token changes. One fixed
fixture: B=8, 128 steps, seed 11. It asserts a nonzero first-step drift, max relative state
drift < 0.05, and that the second half is no more than 2× the first half plus 1e-3. It is a
regression bound for that fixture, not evidence of a general plateau, of prefill/decode
equality, or of any bound on model logits. Earlier 1024-step and prefill-vs-decode tables were
withdrawn in dbb8d00 because they had no checked-in runner (:25-28, design note :214-217).

Adapted from the design note §6; the MTP and backward reasons are restated against RFC #428:

Also not covered: the prefill side. vLLM's bundled FLA chunked kernel rejects fp32 q/k/v
(third_party/flash_linear_attention/ops/chunk.py:213). On SM100 with head_k_dim=128 and a
CUDA ≥ 13 runtime, though, _resolve_gdn_prefill_backend selects FlashInfer by default
(qwen_gdn_linear_attn.py:129-147), so that assert is not on the default prefill path on this
hardware. FlashInfer's SM100 prefill paths reject fp32 q/k/v as well: the non-CP path via
_cutlass_io_dtype (flashinfer/gdn_kernels/blackwell/gdn_prefill.py:66-74) and the CP path
(gdn_cp_prefill.py:736-738); that is from reading flashinfer 0.6.18.post1, not from a run. An
fp32 q/k/v prefill is therefore unavailable on either default backend. The opt-in CuteDSL
backend (qwen_gdn_linear_attn.py:108, :137, :148) was not checked.

• 7b30633: decode-kernel selection under gqa_interleaved_layout (§1); the actual MTP path;
  separate conv and recurrent NULL_BLOCK_ID contracts (§3); softplus not being a
  transcription (§2); validate_data not checking index values.
• 2d895d0: measured agreement tables from the runner (§4).
• 52f7f61: the FP-contraction result (replaced in 610f1b7).
• 7862a12: the mixed-step caveat scoped to what was measured (§5).
• 1d69312: renamed to docs/design/rfc428-c6-gdn-recurrent-replay.md and retitled "RFC [RFC][Qwen3-Next-80B-A3B-Instruct][CUDA/ROCm] VIME/vLLM operator-level train-rollout consistency roadmap, ablation matrix, and integration plan #428
  C6", backward written "RFC [RFC][Qwen3-Next-80B-A3B-Instruct][CUDA/ROCm] VIME/vLLM operator-level train-rollout consistency roadmap, ablation matrix, and integration plan #428 C7", so neither collides with upstream's WS1 C6 ([WS1 Closeout][KV] Direct decode-prefill consistency sweep #272) / C7
  ([WS1 Closeout][KV] Stateful cache path and generate-rescore consistency #273); the §5 mixed-step numbers and the §7 startup failure labelled as reported
  observations, not acceptance evidence; every bound labelled a regression bound, with the
  contract's forward_accuracy/by_op_class/reduction row for scale (§4, :122-127).
• 610f1b7: the two conv mismatch mechanisms, measured by the runner at f8d9399, and the
  recurrent provider's Triton exp/sigmoid as an open item (§4, :169-210).

As of 2026-10-01 no CI path is known to execute these tests, before or after merge.

• On main (32b765e and the current 1968a87), CI-Pipeline's linting fails at
  Run pre-commit hooks, so unit-tests is skipped. Docs fails at both commits too.
• WS1-chain-GPU and ws1-gtest-gpu stop at Configure runpodctl. The cause is step order:
  runpodctl config --apiKey, run as the first runpodctl call on a fresh runner, finds no
  config file (run 36835086246 logs 'runpodctl config' is deprecated then
  error saving config, reproduced locally with v2.14.0 on an empty HOME). ws1-gtest-gpu
  has never succeeded: as of 2026-10-01, 122 runs (69 failed, 40 skipped, 13
  action_required), and its first run (2026-08-19) already failed at this step. A separate
  fix is being prepared.
• gpu-ci (pull_request_target only, gpu-ci.yml:4; label needs-gpu-ci) uses the same
  command (:50) but runs runpodctl version first, which creates the config file, so it got
  past setup on its last run that reached RunPod (2026-09-11, run 34585491929) and then failed
  at pod create. No run since. Its pytest tests/ (ci/run_gpu_ci.sh:133) would also stop at
  the aiter collection error.
• ws1-chain-npu was queued on 1968a87 and cancelled on 32b765e.

If those are fixed, the workflows' scripts would do this with this branch:

1. ci.yml (needs: linting; push and PR to main/test): unit-tests now runs
   tests/test_gdn_state_contract.py on a CPU runner (ci.yml:70), not the check_ file.
2. ws1-gtest-gpu.yml: unchanged by this branch. The goldens fall under its
   rl_engine/kernels/ops/** path, so it triggers for same-repo PRs and on push to main;
   fork PRs skip it (:59). Its script ci/run_ws1_gtest.sh runs neither GDN file.
3. qwen3-next-provider-gpu.yml (new): runs both check_ files (:82) after checking the
   runtime versions and that rl_engine._C loaded from $GITHUB_WORKSPACE (:55-79). It runs
   on same-repo PRs to main (paths :20-25) and on workflow_dispatch, not on push; fork PRs
   get only a notice job (:36-45). It needs a self-hosted runner labelled
   rl-kernel-qwen3-next (:49); whether upstream has one has not been confirmed.
4. gpu-ci.yml: ci/run_gpu_ci.sh runs pytest tests/, which collects
   test_gdn_state_contract.py but not the check_ file. The check_ file would also skip
   there: the script builds on a runpod/pytorch:2.4.0 image with torch 2.4.1 and never
   installs vLLM (run_gpu_ci.sh:19, :193-200).
5. ws1-chain-gpu.yml: push and PR to main/test, no paths filter, skipped for fork
   PRs (:46). Its chain (ci/run_ws1_chain_gate.sh) instantiates each profile's declared
   candidate class by path, not through the registry (rl_engine/alignment/qwen3_dense.py:374-396;
   RMSNormCudaOp for rms_norm); none of this layer's own files are on that path;
   RMSNormCudaOp itself is changed by the PRs below.

• sm_100 (Blackwell), not the sm_90 the project's benchmarks usually use. Per RFC §2.2 no claim
  is made across H100/B200.
• Stacked on feat/cuda-qwen3-next-gated-rmsnorm ([WS1][CUDA][Qwen3-Next] Gated RMSNorm CUDA kernel for the GDN block (RFC #428 stack 3/4) #468, cf68fd9), which is on
  feat/cuda-qwen3-next-c1-norm ([WS1][CUDA][Qwen3-Next] C1: zero-centred RMSNorm references and CUDA kernel (RFC #428 stack 2/4) #467) and fix/registry-cuda-rmsnorm-dispatch ([WS1][CUDA][Qwen3-Next] Restore CUDA dispatch for rms_norm (RFC #428 stack 1/4) #466); all three
  must merge first, and this branch contains their commits. Linear since 2026-10-04 (see Latest
  Status); 395fef8 merges cleanly into the current upstream/test-qwennext (11cac8c,
  git merge-tree --write-tree).

• F5, no operator page. There is no docs/operators/ page for the two goldens; the design
  note docs/design/rfc428-c6-gdn-recurrent-replay.md is the only documentation.
• F15, private imports. The runner loads the check_ file and calls its underscore-private
  helpers and _RECURRENT_BOUNDS (scripts/ws1_gdn_provider_agreement.py:147-181, :267-273, :356-359).
• F16, duplicated chunked sum. _chunked_sum (gated_delta_rule.py:76-94) re-implements
  the fixed 32-wide chunking of shape_invariant_rstd (rl_engine/kernels/ops/pytorch/norm/rms_norm.py:100)
  rather than sharing it.
• F18, function-local imports, e.g. CausalConv1dUpdateOp imported inside test functions
  (check_gdn_recurrent_golden.py:260, :327, :340, :432, :452).
• F19, history in docstrings and the note, e.g. the withdrawn tables
  (check_gdn_recurrent_golden.py:25-28) and "an earlier version of this note" (design note
  :149-151).

None of these changes behavior. They are left for follow-ups; this is the top of the stack, so
a follow-up PR is the natural place.